Lidar system with projection capability

ABSTRACT

A combined sensing and projection apparatus includes a LIDAR sensor, a scan module, a unified processor and a mechanical mounting. The LIDAR sensor is configured to detect light and sense a physical property of an object or environment. The scan module has a micro-electromechanical system (MEMS) mirror configured to deflect one or more laser beams to project vector graphic content related to the physical property of the object. The unified processor is configured to reduce processing delays by fusion of processing of both vector graphic content and determination of the physical property of the object. The scan module and sensor are attached to the mechanical mounting.

CLAIM OF PRIORITY

This application claims the priority benefit of U.S. Provisional PatentApplication No. 63/293,316 filed Dec. 23, 2021, the entire disclosuresof which are incorporated herein by reference.

BACKGROUND

Sensors and actuators are essential parts of smart technology that havebeen widely used today in many aspects of human activities includingindustrial automation and transportation. We are also witnessing massiveuse of sensors in commercial, consumer, medical and many other products.However, in many of these applications sensors are still far fromachieving optimal performance, and very often what is lacking is timelydisplay of results or messages intended for users. This is particularlytrue in robotics and similar working environments that requireinteraction between machines and humans. There is an obvious need forefficient machine-human interaction and intention communication toimprove safety. Today such solutions are in their infancy.

In addition, almost many useful machines are typically battery operated,consequently sub-systems that augment the capabilities of such machinesmust be low-power solutions. In the case of airborne drones, it is alsocritical that solutions are ultra-light weight and of small form factor.

Prior work has been presented in which robots are equipped with varioussensors including inertial, optical, ultrasonic, etc. to aid the machineto perform its tasks more efficiently. However, only recently have therebeen innovations towards addressing the specific need of human-robotinteraction. Existing solutions are half measures where cameras andLIDAR sensors are able to detect and identify objects within thesensor's field of view, but the intent to communicate effectively withthe user is typically based on flashing bulbs, rotational lights, orscreens and LED based projectors which are very bulky and energyinefficient.

It is within this context that aspects of the present disclosure arise.

OBJECTIVES

The objective of the work that led to aspects of the present disclosurewas to develop and demonstrate a sensing system working alongside withprojecting capabilities based on an optical-MEMS Mirror steeringtechnology. Introducing MEMS Mirror-based solutions such as vectorgraphics laser projection (VGLP) for machine-human interaction bringssignificant improvement in performance and safety. 1VIEMS Mirror basedsolutions offer systems with lowest power consumption, lowest weight,small form factor, and low cost, as well as high contrast programmableand animated messaging that is far superior to flashing bulb or rotatinglight. The present disclosure describes solution having a sensing systemand projection system fused together at an operational level. Thesensing system could be based on any suitable sensor such astemperature, pressure, humidity, radar, or an electromagnetic radiationsensor such as an optical sensor based sensing that can rely on thereflected brightness, time of flight, imaging or other optical sensingmethodologies. The projection system uses vector graphics projectionbased on a MEMS Mirror to provide high contrast information in real timeon any surface in daytime and night conditions. The fusion of sensingand projection systems allows for fast reaction, and immediate feedbackto user which could be done in different ways, by projecting data,projecting warning signs, giving directions, highlighting differentpaths, etc. Also, multiple VGLP projectors or different colors from asingle VGLP projector can be used to enrich communications and improvequality of information. For example, different color can signifyproximity to danger, or different colors can be used to mark differentset of data, or designated to specific user, etc.

With single fused unit the process of sensing or detecting andprojecting data in a timely manner, in a different form and shape,without delay, greatly improves safety and allows for fast correctiveaction when needed.

An additional objective was to develop a low size, low weight, and lowpower solution that will be very attractive for use in battery operatedmachines and instruments. Electrostatic driven MEMS Mirrors inherentlylead to low power scanning solutions, and the MEMS mirror scanningtechnology allows the reduction of size and weight of the solution. Thecomplete low cost and reduced weight and size solution is veryattractive feature for many applications including in drones where thepayload of each part of the system included as a part of drone is ofparamount importance. These features are also important for otherapplications such as ADAS applications in automotive market, or safetyin smart city applications.

SUMMARY

A new system that comprises of two main sub-systems, a sensing systemand a vector graphics laser projector fused together is presented. Thesensing system could be based on any electromagnetic radiation sensor orother type of sensor. Example sensors include, without limitation,temperature, pressure, humidity, radar, or optical sensor. Informationfrom sensors may be used by a processor to determine a physical propertyof an object or environment. The object may be an aspect of theenvironment or part of the environment for example and withoutlimitation, ambient pressure, ambient sound, a floor, a wall, a personetc. The physical property may be for example and without limitation; 3dimensional relative location, 2 dimensional relative location,geospatial location, speed, acceleration, distance, pressure level,sound level, temperature, reflectance, absorption, planarity, luminance,length, width, height, facial features, human presence, object presence,classification/identity (object or facial recognition), etc.

The sensor may be an electromagnetic radiation sensor system that uses alight source that is steered using a MEMS mirror over a field of view(FoV) and the same or additional MEMS mirror is used to image back thereflected light to generate a 3D point cloud, based on the return signalbrightness, time of flight, or additional optical sensing methodologies.The vector graphics laser projection (VGLP) system preferably includes aMEMS mirror-based laser scanner for projecting vector graphic contentusing one or more visible laser beams to produce high brightness andhigh contrast projections. When sensing and projection functions arefused together the resulting system is able to image over the FoV,generate a 3D point cloud using the sensor, or get other informationrelated to environment, and interact with the environment by projectingcontent using the VGLP.

Systems of the type described herein can be used in various markets suchas commercial, consumer, industrial, automotive, etc. A good example ofapplication is in robotics, where the system can be used for sensing theactively changing FoV and interacting with users, navigation, warningthe robot's surroundings, and much more.

Both of the subsystems, sensing system and projection system, have beenindividually demonstrated as separate products, such as the SyMPL 3DLidar from Mirrorcle Technologies Inc. (MTI) of Richmond, Calif. for thesensor of the environment and the Playzer from MTI for the VGLP system.Both subsystems use MEMS mirrors to scan over their respective FoVs andare built on a common technical platform at the hardware and softwarelayers, using MEMS controllers based on MCUs, and software APIs. Thehardware and software allow for the systems to be fully programmable andreconfigurable, with the choice of scan FoV, frame rate, resolution, andso on, for the optical sensor. The content projected by the VGLP systemcan be any arbitrary vector graphic shape pattern, symbol, text, numeralor other character that can be adjusted in projection size, refresh rateand placement within the overall FoV.

Furthermore, MTI has already demonstrated full fusion of these twosubsystems at the API level. The fusion enables the system to zoom inand perform higher resolution scans on the optical sensor side anddirect the Playzer's content projection to highlight data from sensor orprojects warning in specific areas, or address or interact withindividual person or thing.

The fused system is designed to operate under 2 W of power, with somevariation depending on the total Playzer's projection output brightness.The system is designed to fit within a small volume of 200 cm3 orsmaller and weighing under 200 g. The overall small size and weight,with the low power consumption makes this system ideal for robotics anddrone applications, and most other battery-operated applications incommercial and industrial spaces. The system is equally well suited forADAS applications in automotive market as well as many otherapplications.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1B illustrates a schematic overview of an apparatus accordingto aspects the present disclosure having a combination of opticalsensing and laser projection systems and their respective shared fieldsof view (FoV).

FIG. 2 illustrates a schematic overview of an apparatus according toaspects of the present disclosure having a combination of opticalsensing and laser projection systems and their respective fields of view(FoV) that do not overlap.

FIG. 3 illustrates the reconfigurability of a laser projection system'sscan and FoV according to aspects of the present disclosure.

FIG. 4 illustrates the reconfigurability of an optical sensing system'sscan and FoV according to aspects of the present disclosure.

FIG. 5A-5B depicts an apparatus according to aspects of the presentdisclosure in which two or more projection systems are arrayed togethereither horizontally (FIG. 5A) or vertically (FIG. 5B) to achieve acombined larger FoV.

FIG. 6A-6B depicts an apparatus according to aspects of the presentdisclosure in which two or more optical sensing systems are arrayedtogether either horizontally (FIG. 6A) or vertically (FIG. 6B) toachieve a combined larger FoV.

FIG. 7A depicts variations of combining multiple optical sensing systemsand laser projection systems to create a combined larger FoV for bothsensing and projection.

FIG. 7B depicts variations of combining multiple optical sensing systemsand laser projection systems to project in multiple wavelengths for thelaser projection.

FIG. 8 illustrates a schematic overview of an apparatus according toaspects of the present disclosure with sensor and projection systemsrunning from the same processor at the hardware level.

FIG. 9 illustrates a schematic overview of an apparatus according toaspects of the present disclosure working in conjunction with anexternal camera or other sensor(s).

FIG. 10A and FIG. 10B illustrate an Autonomous Mobile Robot (AMR)according to aspects of the present disclosure, sensing obstacles intheir path and projecting warnings and information in front of therobot.

FIG. 11A and FIG. 11B illustrate an Autonomous Mobile Robot (AMR) anapparatus according to aspects of the present disclosure, sensingobstacles in their path and projecting warnings and information in frontof the robot.

FIG. 12 illustrates a drone an apparatus according to aspects of thepresent disclosure, sensing obstacles in its path or ground andprojecting warnings and information in its landing zone.

FIG. 13A and FIG. 13B are block diagrams of the technical flow for thepresent invention In which FIG. 13A depicts a modular approach withindividual electronics and controllers for the VGLP and sensing systemintegrated in common enclosure and API level and FIG. 13B depicts aunified hardware and software platform integrated in common enclosure,common MCU, and API level. Here a single processor interprets theoptical sensing information, interfaces with a host system, and projectsthe information to the user

FIG. 14A shows a monochrome VGLP system mounted on a robot vacuumcleaner, displaying content in a single color according to an aspect ofthe present disclosure.

FIG. 14B shows a full color RGB VGLP system mounted on a robot vacuumcleaner, displaying content in various colors using a combination of RGBlasers according to aspects of the present disclosure.

FIGS. 15A-15C depict apparatus according to aspects of the presentdisclosure displaying content in different colors simultaneously toconvey additional meanings such as warnings, caution, etc. The multiplecolors can also be used in conjunction with each other to projectmultiple pieces of information (FIG. 15C).

FIG. 16A depicts a moving robot equipped with an apparatus according toaspects of the present disclosure using a sensing system to measuredistance and angle relative to the wall while moving and VGLP system tolive-stream that data and project on the wall.

FIG. 16B depicts an apparatus according to aspects of the presentdisclosure in which a VGLP system works wirelessly via Bluetooth todisplay information.

FIGS. 17A and 17B depict a cleaning robot outfitted with an apparatusaccording to aspects of the present disclosure that uses a LIDAR sensorto detect an obstacle, specifically a person's legs, and then project agreeting with a full color (RGB) VGLP system. The information isprojected and customized with respect to person so that it is visibleand readable by the person the robot detected.

FIG. 17C depicts a cleaning robot with an apparatus according to aspectsof the present disclosure that uses an ultrasonic sensor system todetect an obstacle and the sensor is combined with the RGB laserprojector (VGLP system)—in this case displaying a company logo.

FIG. 18A shows a prototype of a system having an apparatus according toaspects of the present disclosure that combines a camera with a VGLPsystem.

FIG. 18B shows a superposition of camera FoV and VGLP system field ofregard for an apparatus according to aspects of the present disclosure.Within the overlapped area is the calibrated Field of Regard regionwhere the two are fully calibrated with respect to each other.

FIG. 18C-18D show a prototype of a system having an apparatus accordingto aspects of the present disclosure that combines a camera with a VGLPsystem where camera detects a certain feature in an image and the VGLPlaser system projects a beam to that point.

FIG. 19A-19C show a prototype of a system having an apparatus accordingto aspects of the present disclosure that combines a camera with a VGLPsystem where camera detects a person and the VGLP system projects someentertaining content in front of a person's feet.

FIG. 20 is a schematic diagram of an apparatus according to aspects ofthe present disclosure.

DETAILED DESCRIPTION

Although the following detailed description contains many specificdetails for the purposes of illustration, anyone of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within the scope of the invention. Accordingly,the exemplary embodiments of the invention described below are set forthwithout any loss of generality to, and without imposing limitationsupon, the claimed invention.

In the interest of clarity, not all of the routine features of theimplementations described herein are shown and described. It will beunderstood by those skilled in the art that in the development of anysuch implementations, numerous implementation-specific decisions must bemade in order to achieve the developer's specific goals, such ascompliance with application- and business-related constraints, and thatthese specific goals will vary from one implementation to another andfrom one developer to another. Moreover, it will be appreciated thatsuch a development effort might be complex and time-consuming, but wouldnevertheless be a routine undertaking of engineering for those ofordinary skill in the art having the benefit of the present disclosure.

According to an aspect of the present disclosure shown in FIG. 1A, asystem consists of a Vector Graphics Laser Projection (VGLP) system 102and sensing system (SENS) 103, coupled together on a mechanical mounting101. The VGLP system is a MEMS mirror-based laser projection system. Thesensor could be any type of electromagnetic radiation sensor or othertype of sensor Example sensors include, without limitation, temperature,pressure, humidity, radar, or optical sensor. For some of sensors thefield of view may be a single point, meaning that sensor measures onepoint parameter such as temperature, or humidity, while other sensorshave an effective region or volume in which they sense environmentalparameters (e.g. field of view for an optical sensor or radar orultrasound), or distance (laser range finder) may also be the sensingsystem in some implementations. FIG. 1A shows the case for the sensorthat has field of view. In some implementations, the sensing system 103and VGLP system 102 may be arranged such that their apertures are fullycoaxial and the FOV for either of them can be adjusted, aligned, and/orexpanded by commonly shared optics.

The projector and sensor are mounted in a way that fixes therelationship of their respective fields of view (FoVs), with 104defining the VGLP system's FoV and 105 defining the sensor's FoV. Themounting 101 provides for them to have fixed FoVs and they are mountedto maximize their shared FoV 106 where the system can both sense andproject. The system is coordinated and controlled via a unified softwareplatform 107. According to aspects of the present disclosure theprojector and sensor may be attached to the mounting 101 by eitherpermanent means, e.g., welding, epoxying, riveting, and the like, or byremovable means, e.g., screws, bolts, magnets, vacuum cups, Velcro, andthe like.

Information from sensors may be used by the unified software platformrunning on a processor to determine a physical property of an object orenvironment. The object may be an aspect of the environment or part ofthe environment for example and without limitation, ambient pressure,ambient sound, a floor, a wall, a person, a face, etc. The physicalproperty may be for example and without limitation; 3 dimensionalrelative location, 2 dimensional relative location, geospatial location,speed, acceleration, distance, pressure level, sound level, temperature,reflectance, absorption, planarity, luminance, length, width, height,facial features, human presence, object presence,classification/identity (object or facial recognition), etc.

FIG. 1B shows the same basic implementation shown in FIG. 1A, but withthe sensing system 103 FoV being a different resolution than the VGLPsystem 102. This different FoV 109 is shown as a smaller FoV. Theresulting overlap still creates a shared FoV 106, where the system canboth sense and project.

FIG. 2 shows a system comprising a VGLP system 202 and a sensing system203, coupled together on a mechanical mounting 201. The projector andsensor are mounted in a way to fix the relationship of their respectivefields of view (FoVs), with 204 defining the VGLP system's FoV and 205defining the sensor's FoV. The substrate provides for them to have fixedFoVs and they are mounted to sense and project in their respectiveareas, with no overlap in their FoVs. The system is coordinated andcontrolled via a unified software platform 207. It is understood thatthe sensing system may not have a sharply defined region of sensing asillustrated and that the region or volume within which it sensesenvironmental parameters is not specifically a field of view. However,for the purposes of illustration of the invention it is shown as aconical or pyramidal volume typically seen in optical sensors such aslidar or radar sensors. Similarly, in embodiments where sensor isanother type of electromagnetic radiation sensor such as, for exampleand without limitation a thermal camera or a camera for any range ofwavelengths (visible range, NIR, IR, MWIR, LWIR, etc.) Additionally,regions of sensing may be different.

FIG. 3 shows the reconfigurability of the VGLP system 302. The systemcan reconfigure the arbitrary scan from the MEMS mirror to zoom in andaddress specific regions 307, within the total addressable FoV 304. Thismay also be termed as the Field of Regard (FoR) of the projector inwhich it can address select objects or targets with one or more laserbeams with one or more wavelengths. The programmability of the MEMSmirror in the VGLP allows for the scan to be reconfigurable, and able toproject any arbitrary shapes, glyphs, text, etc.

FIG. 4 shows the reconfigurability of the sensing system 403. In someembodiments the sensing system may also have a reconfigurable region orvolume of sensing (“field of view”). For example, a MEMS Mirror based 3Dlidar could be used as the sensing system in which the MEMS mirror canbe programmed to scan and to zoom in and scan specific regions 408,within the total addressable FoV 405. In such an embodiment where thesensing system is a 3D Lidar, the programmability of the beam steeringelement in the sensing system allows for the scan to be reconfigurable,and able to image specific objects, add additional points and resolutionto the scan, variable refresh rates, etc. The same holds in embodimentwhere sensor system is a radar system.

In an implementation where the sensing system is a camera VGLP systemcan also have different FoV than camera and it may have a different lenswhich can be controlled to zoom and choose the FoV.

FIG. 5A shows that VGLP systems 501, 503 may be modular, and that anarbitrary number of them (indicated at 502) can be arrayed togetheralong the X-axis. There are several reasons to array multiple VGLPsystems in embodiments. Firstly, they can be arrayed in such a way thatthey increase the total projection FoV 504, 505. Second reason forarraying multiple projectors may be to project more complex contentwithin the same shared FoV. Also, multiple individual VGLP systems canbe arrayed and configured to project in different wavelengths. Anycombination of these arraying concepts may also be utilized. Forexample, one projector may project slightly to the left, anotherprojector to the right, achieving a larger total addressable projectionFoV, and then a third projector may be added but with a differentwavelength to project different information within that combined FoV ofthe first and second projector.

FIG. 5B shows that VGLP systems 501, 503 are modular, and that anarbitrary number of them (indicated at 502) can be arrayed togetheralong the Y-axis to increase the projection FoV 506, 507. Thecombination of the systems in both X and Y-axes allows for the system tobe modular and scalable for any application. This array of modules canbe configured to work to project a single complex content, or can beconfigured as multiple individual systems that project in differentwavelengths, or can be configured as multiple individual systems withFoVs that combine into one larger overall FoV, etc.

FIG. 6A shows that the setup of the overall sensing system may also bemodular. For example, multiple sensing systems 601, 603 can be arrayedtogether in arbitrary numbers (as indicated at 602) along the X-axis tocombine the sensing FoVs 604, 605 into a larger overall FoV or sensingregion or volume. This array of modules can be configured to work toaddress a larger area, or multiple individual subsections that can sensewith different resolution, areas of regard, etc. In some implementationsthe overall sensing system is a combination of multiple types ofsensors, arrayed together to observe more environmental parameters orinputs. For example 601 may be a radar sensor and 603 may be a camerasensor. There may be additional sensors arrayed along those two to addmore sensing functionalities or to increase the functionality of any oneof those.

FIG. 6B shows that sensing systems 601, 603 are modular, and can bearrayed together in arbitrary numbers, as indicated at 602, along theY-axis to increase the projection FoV 606, 607. The combination ofsystems in both X and Y-axes allows for the system to be modular andscalable for any application. This array of modules can be configured towork to address a larger area, or multiple individual subsections thatcan sense with different resolution, areas of regard, etc.

FIG. 7A and FIG. 7B show that VGLP systems 701, 703 and sensing systems706, 708 are modular and can be arrayed together in arbitrary numbers,as indicated at 702, along the X and Y axes to increase their overallprojection and sensing FoVs. The VGLP systems' FoVs are defined by 704and 705. The sensing system's FoVs are defined by 707 and 709. Thecombination of both the systems in X and Y-axes allows for the system tobe modular and scalable for any application. This array of projectionand sensing modules can be configured to work to sense a larger area, ormultiple individual subsections with different resolution, areas ofregard, and be able to project content out over that FoV, to highlightspecific areas using different wavelengths, interact with other elementsin the FoV, etc.

FIG. 8 illustrates an apparatus according to aspects of the presentdisclosure in which a system includes a projector (e.g., VGLP) system802 and an sensing system 803, coupled together in mechanical housing801. The projector and sensor systems are mounted in a way that fixesthe relationship of their respective fields of view (FoVs), with 804defining the VGLP system's FoV and 805 defining the sensor's FoV. Thesubstrate provides for them to have fixed FoVs and they are mounted tomaximize their shared FoV 806 where the system can both sense andproject. The system is coordinated and controlled via a unified hardwareprocessor 807 (MCU). The hardware processor controlling both sub-systemsis located in the same mechanical housing 801 to simplify the userinterface, reduce any processing delays as they are all handled on thesame hardware processing layer.

FIG. 9 illustrates an implementation similar to that depicted in FIG. 7, where a system consists of a (VGLP) system 902 and a sensing system903, coupled together in mechanical housing 901. The projector andsensor are mounted in a way that fixes the relationship of theirrespective fields of view (FoVs), with 904 defining the VGLP system'sFoV and 905 defining the sensor's FoV. The substrate provides for themto have fixed FoVs and they are mounted to maximize their shared FoV 906where the system can both sense and project. The system is coordinatedand controlled via a unified hardware processor (MCU) 907. In thissetup, an additional (auxiliary) sensor or camera (AUX) 908 is placedrelative 910 to the sense and scan system 901. This additional sensor orcamera has a separate field of view 909 that can be used for adding asecondary and independent sensing system for safety or reliability ofthe present invention that may be required for certain industryapplications such as automotive, industrial, etc.

FIG. 20 depicts a unified processor computing system with VGLP andsensor according to aspects of the present disclosure. The system mayinclude a computing device 2000, such as single board computer (SBC),system on chip (SOC), microcontroller or similar, coupled to at one ormore scan modules 2002 and one or more sensor systems 2022. The scanmodule 2002 may include a light projector such as vector graphics laserprojection (VGLP) system which is a

MEMS mirror-based laser scanner for projecting vector graphic contentusing one or more visible laser beams to produce high brightness andhigh contrast projections. The sensor system 2022 may include anyelectromagnetic radiation sensor or other type of sensor. Examples ofsensors include, without limitation, temperature, pressure, humidity,radar, or optical sensor. The optical sensor may be an electromagneticradiation sensor system that uses a light source that is steered using aMEMS mirror over a FoV and the same or additional MEMS mirror is used toimage back the reflected light onto a light sensing element to generatea 3D point cloud, based on the return signal brightness, time of flight,or additional optical sensing methodologies. An optical time of flightsensor may be a specialized time of flight sensor such as a pulsedoptical time of flight sensor, a Frequency Modulated Continuous Wave(FMCW) optical time of flight sensor or an Amplitude ModulatedContinuous wave (AMCW) optical time of flight sensor. In someimplementations, the scan module(s) 2002 and/or the sensor systems 2022may each include an integrated processor (not shown) that controlscertain aspects of their respective operation. For example, the sensorsystem(s) may include a microprocessor or microcontroller that convertssignals from a sensor element from one data format to another, e.g.,from analog to digital, and performs preliminary processing, such asscaling or calibration on the converted data. The scan module(s) mayinclude a microprocessor or microcontroller that converts data in a textor image format into commands that control one or more laser sources andone or more scanning mirrors. The integrated processors may communicatewith the computing device 2000, which can fuse the operation of the scanmodule(s) 2002 and sensor system(s) 2022 at firmware level or anapplication program interface (API) level.

The one or more sensor systems 2022 and one or more scan modules 2002may be coupled to or otherwise attached to a common mechanical mounting2023. The sensor system(s) and scan module(s) may be located withinphysical proximity to each other on the mechanical mounting 2023.Additionally a Processor 2003 and other components of the computingsystem 2000 may be located on the mechanical mounting 2023 and may bewithin physical proximity of the sensor system(s) 2022 and scanmodule(s) 2002. The physical proximity of the processor 2003 and othercomponents of the computing system with the at least one scan module2002 and at least one sensor 2022 may further reduce processing delays.

The computing device 2000 may include one or more processor units 2003,which may be configured according to well-known architectures, such as,e.g., single-core, dual-core, quad-core, multi-core,processor-coprocessor, cell processor, and the like. The computingdevice may also include one or more memory units 2004 (e.g., randomaccess memory (RAM), dynamic random access memory (DRAM), read-onlymemory (ROM), and the like).

The processor unit 2003 may execute one or more programs, portions ofwhich may be stored in the memory 2004 and the processor 2003 may beoperatively coupled to the memory, e.g., by accessing the memory via adata bus 2005. The programs may be configured to display vector graphiccontent 2008 with the scan module 2002 using Scanner controls 2021 tocommand portions of the scanner such as lasers and MEMS mirrors.Additionally the Memory 2004 may contain programs that determine aphysical property of an object 2009 from sensor information 2010. Thesensor information 2010 may be information from the one or more sensors2022 and the sensor information 2010 may interpret, filter or format theinformation from the one or more sensors to place it in a formcompatible with other systems and/or programs.

The sensor information and vector graphics may also be stored as data2018 in the Mass Store 2015. The processor unit 2003 is furtherconfigured to execute one or more programs 2017 stored in the mass store2015. The programs 2017 (or portions thereof) may be configured, e.g.,by appropriate programming, to determine a physical property of anobject from sensor information or control the scanner which may be readdirectly from the mass store 2015 or loaded into memory 2004 from themass store.

The computing device 2000 may also include well-known support circuits,such as input/output (I/O) 2007, circuits, power supplies (P/S) 2011, aclock (CLK) 2012, and cache 2013, which may communicate with othercomponents of the system, e.g., via the bus 2005. The computing devicemay include a network interface 2014. The processor unit 2003 andnetwork interface 2014 may be configured to implement a local areanetwork (LAN) or personal area network (PAN), via a suitable networkprotocol, e.g., Bluetooth, for a PAN. Additionally the network interfacemay be an interconnect to other devices or computer systems such as a,Peripheral component interconnect (PCI), PCI express, serial interface,universal serial bus, or the like. The computing device may optionallyinclude a mass storage device 2015 such as a disk drive, CD-ROM drive,tape drive, flash memory, or the like, and the mass storage device maystore programs and/or data. The computing device may also include a userinterface 2016 to facilitate interaction between the system and a user.The user interface may include a display, e.g., a monitor, Televisionscreen, speakers, headphones or other devices that communicateinformation to the user. In addition, the user interface may include oneor more input devices, e.g., buttons, switches, a keyboard, joystick,trackball, touch pad, touch screen, microphone, etc. The computingsystem 2000 may communicate with a network 2020 through the networkinterface.

The network 2020 may be one or more other computers or devicesconfigured to communicate with the computing device 2000. The files,sensor information and commands may be exchanged over the network 2020between the computing systems and other computing systems and devicesconnected to the network.

FIG. 10A shows the use of an apparatus according to aspects of thepresent disclosure on an Autonomous Mobile Robot (AMR) 1001. In thisimplementation, a sensing system 1002 has a sensing field of view 1004to detect obstacles 1007 within the robot's path. A VGLP system 1003projects a warning message 1006 within the projector's field of view1005.

FIG. 10B shows another example of use of the Autonomous Mobile Robot(AMR) 1001 of FIG. 10A. The sensing system 1002 has a sensing field ofview 1004 to detect any obstacles such as a wall 1007 within the robot'spath. The VGLP system 1003 projects a warning and distance message 1006within the projector's field of view 1005 onto the target wall 1007.

FIG. 11A shows another example of use of an apparatus according toaspects of the present disclosure on an Autonomous Mobile Robot (AMR)1101. A sensing system 1102 has a sensing field of view 1104 to detectany obstacles 1107 within the robot's path. A VGLP system 1103 projectsa highlighted path with various colors warning the user as they approachcloser it is more dangerous. The colors could, for example, indicate asafe region 1110, a warning region 1109 and a danger region 1108 indifferent colors. In this example, these regions are all within theprojector's field of view 1105.

FIG. 11B shows another example of use of an apparatus according toaspects of the present disclosure on an Autonomous Mobile Robot (AMR)1101. The sensing system 1102 has a sensing field of view 1104 to detectany obstacles 1107 within the robot's path. The VGLP system 1103projects a highlighted path it is traveling in with flashing arrows. Thecolors and size may scale as it gets further away from the robot toalert users as they approach the robot, with a larger arrow 1109 furtherfrom the robot, and a smaller arrow closer to the robot 1108. Thesearrows are all within the projector's field of view 1105 in theillustrated example.

FIG. 12 shows another example of use of an apparatus according toaspects of the present disclosure on a drone 1201. A sensing system 1202has a sensing field of view 1204 to detect any obstacles on the ground1207 within the robot's path. The VGLP system 1203 projects a warningand distance message 1206 within the projector's field of view 1205 ontothe ground landing area 1207.

FIG. 13A and FIG. 13B are block diagrams of the technical flow for thepresent invention. FIG. 13A depicts a more modular approach withindividual electronics and controllers for the VGLP and sensing system.Both these individual systems would interface with a unified APIsoftware platform layer and with a host system via a softwareapplication.

FIG. 13B describes an integrated hardware and software platform where asingle processor interprets the optical sensing information, interfaceswith a host system, and projects the information to the user in the VGLPdisplay system. The single processor is able to process the 3D positioninformation, and prepare the projection content onboard at the hardwarelayer, but can also communicate with the higher unified API softwareplatform layer and application on the host computer.

FIG. 14A shows a monochrome VGLP system mounted on a robot vacuumcleaner, displaying content in a single color, e.g., green. TheMonochrome VGLP system is a single-color projector, where the hardwareinstalled has only one laser module inside.

FIG. 14B shows a full color RGB VGLP system mounted on a robot vacuumcleaner, displaying content in various colors using a combination of RGBlasers. The RGB VGLP system is a full color projector, where thehardware installed has three lasers combined together to form a singlebeam output to be scanned and projected from the module.

FIGS. 15A-15C depicts an implementation in which the same content can bedisplayed in different colors to indicate additional meanings of themessages being displayed such as warnings, caution, etc. FIG. 15A showsa green and red color Monochrome VGLP projector displaying information.FIG. 15B shows a red and green color Monochrome VGLP projectordisplaying information in graphical manner. FIG. 15C shows in red andgreen information projected on two separate surfaces using the twoseparate Monochrome VGLP systems

FIG. 16A shows a full robot system using an optical sensor to measuredistance and angle to a wall and streaming and projecting thatinformation visually using a VGLP system. The robot can navigate using3D point cloud information gathered by the optical sensor, and displayinformation as needed to any users nearby. In this case, the VGLP isused for showing to user information of distance and angle to anyobstacles in front of the robot.

FIG. 16B depicts an apparatus according to aspects of the presentdisclosure in which a VGLP system works wirelessly via Bluetooth todisplay information. In the illustrated example, the system communicateswith a cellular phone to project content. Alternatively, the VGLP systemmay communicate with any suitable Bluetooth-capable device in a likemanner to project content generated with, stored on, or selected by thedevice.

FIG. 17A and FIG. 17B show a cleaning robot using a lidar sensor todetect an obstacle, specifically a person's legs, and then project agreeting with a full color (RGB) VGLP system. The information isprojected and customized with respect to person so that it is visibleand readable by the person the robot detected.

FIG. 17C shows a cleaning robot that is using ultrasonic sensor systemto detect an obstacle and the sensor is combined with the RGB laserprojector (VGLP system)—in this case displaying a company logo.

FIG. 18A shows a prototype of a system which combines a camera with aVGLP system. In this case the integration is done at API level.

FIG. 18B shows a superposition of camera FoV and VGLP system field ofregard. Within the overlapped area is the calibrated Field of Regardregion where the two sub-systems are fully calibrated with respect toeach other.

FIGS. 18C and 18D show a prototype of a fused system which combines acamera with a VGLP system where camera detects a certain feature in animage and the VGLP laser system augments information by projecting abeam to that point.

FIG. 19A-19C show a prototype of a system which combines a camera with aVGLP system where camera detects a person and the VGLP system projectssome entertaining content or information that is of interest to person.The information has been projected in front of the person's feet.

BENEFITS

There are multiple benefits of apparatus of the type described in thepresent disclosure.

A single fused system that combines the process of sensing or detectingwith projecting into a seamless process where data is processed locallyin a timely manner and immediately projected in a different form andshape, without delay can greatly improve safety and allow for fastcorrective action when needed.

The fusion of sensing and projection systems allows for fast reaction,and immediate feedback to user which could be done in different ways, byprojecting data, projecting warning signs, giving directions,highlighting different paths, etc. Also, fusing sensing and multipleVGLP projectors or different colors from a single VGLP projector can beused to enrich communications and improve quality of information.Projecting in different colors or in multiple spaces simultaneouslyaugments information further. For example, different colors can signifyproximity to danger, or different colors can be used to mark differentset of data, or separate color can be allocated to address specificuser, or different set of data can be projected simultaneously intoadjacent space for comparative viewing, etc.

With a single fused unit the processing of data can happen locally in acommon MCU that is shred therefore using a shared MCU may reduce thenumber of components, which may reduce the size and weight of the totalsolution. Using one MCU instead of two may reduce the power consumption.This combined benefits with already low power consumption associatedwith MTI MEMS mirrors creates the most competitive fusion solution.

The overall combination of low power consumption, low weight, and smallsize leads also to low cost of the fused solution. All of these are veryattractive features for many applications including in robots, drones,ADAS applications in automotive market, safety in smart cityapplications, and many other areas.

While the above is a complete description of the preferred embodimentsof the present invention, it is possible to use various alternatives,modifications, and equivalents. Therefore, the scope of the presentinvention should be determined not with reference to the abovedescription but should, instead, be determined with reference to theappended claims, along with their full scope of equivalents. Anyfeature, whether preferred or not, may be combined with any otherfeature, whether preferred or not. In the claims that follow, theindefinite article “A” or “An” refers to a quantity of one or more ofthe item following the article, except where expressly stated otherwise.The appended claims are not to be interpreted as includingmeans-plus-function limitations, unless such a limitation is explicitlyrecited in a given claim using the phrase “means for”. Any element in aclaim that does not explicitly state “means for” performing a specifiedfunction, is not to be interpreted as a “means” or “step” clause asspecified in 35 USC § 112, ¶ 6.

PUBLICATION REFERENCES

-   [1] Milanović, V., Castelino, K., McCormick, D., “Highly Adaptable    MEMS-based Display with Wide Projection Angle,” 2007 IEEE 20th    International Conference on Micro Electro Mechanical Systems (MEMS),    Hyogo, Japan, Jan. 21-25, 2007, pp. 143-146-   [2] Milanović, V., “Linearized Gimbal-less Two-Axis MEMS Mirrors,”    2009 Conference on Optical Fiber Communication, San Diego, Calif.,    USA, Mar. 22-26, 2009, pp. 1-3-   [3] Miner, A., Milanović, V., “High Temperature Operation of    Gimbal-less Two-Axis Micromirrors,” 2007 IEEE/LEOS International    Conference on Optical MEMS and Nanophotonics, Hualien, Taiwan, Aug.    12-Jul. 16, 2007, pp. 91-92-   [4] Milanovié, V., Kasturi, A., Hachtel, V., “High brightness 1VIEMS    mirror based head-up display (HUD) modules with wireless data    streaming capability,” Proc. SPIE 9375, MOEMS and Miniaturized    Systems XIV, 93750A, San Francisco, Calif., USA, Feb. 27, 2015-   [5] Kasturi, A., Milanović, V., Lovell, D., Hu, F., Ho, D., Su, Y.,    Ristic, Lj., “Comparison of MEMS Mirror LiDAR Architectures,” Proc.    SPIE 11293, MOEMS and Miniaturized Systems XIX, 112930B, San    Francisco, Calif., Feb. 28, 2020-   [6] Wengefeld, T., Hochemer, D., Lewandowski, B., Köhler, M., Beer,    M., Gross, H., “A Laser Projection System for Robot Intention    Communication and Human Robot Interaction,” 2020 29th IEEE    International Conference on Robot and Human Interactive    Communication (RO-MAN), Virtual Conference, Aug. 31-Sep. 4, 2020,    pp. 259-265-   [7] Vogel, C., Walter C., Elkmann, N., “A Projection-based Sensor    System for Safe Physical Human-Robot Collaboration,” 2013 IEEE/RSJ    International Conference on Intelligent Robots and Systems (IROS),    Tokyo, Japan, Nov. 3-7, 2013, pp. 5359-5364-   [8] Andersson, N., Argyrou, A., Nagele, F., Ubis, F., Campos, U E.,    Ortiz de Zarate, M., Wilterdink, R., “AR-Enhanced    Human-Robot-Interaction—Methodologies, Algorithms, Tools,” 6th CIRP    Conference on Assembly Technologies and Systems (CATS), Procedia    CIRP, Vol. 44, 2016, pp. 193-198-   [9] Vogel, C., Walter C., Elkmann, N., “Exploring the possibilities    of supporting robot-assisted work places using a projection-based    sensor system,” 2012 IEEE International Symposium on Robotic and    Sensors Environments Proceedings, Magdeburg, Germany, Nov. 16-18,    2012, pp. 67-72-   [10] Sheridan, T B., “HUMAN-ROBOT INTERACTION: STATUS AND    CHALLENGES,” Human Factors, Vol. 58, no. 4, June 2016, pp. 525-532-   [11] Zaeh, M F., Vogl, W., “Interactive Laser-Projection for    Programming Industrial Robots,” 2006 IEEE/ACM International    Symposium on Mixed and Augmented Reality, Santa Barbara, Calif.,    USA, Oct. 22-25, 2006, pp. 125-128-   [12] Ali, R., Liu, R., He, Y., Nayyar, A., Qureshi, B., “Systematic    Review of Dynamic Multi-Object Identification and Localization:    Techniques and Technologies,” IEEE Access, Vol. 9, pp. 122924-122950-   [13] Vogel, C., Poggendorf, M., Walter, C., Elkmann, N., “Towards    Safe Physical Human-Robot Collaboration: A Projection-based Safety    System,” 2011 IEEE/RSJ International Conference on Intelligent    Robots and Systems, San Francisco, Calif., USA, Sep. 25-30, 2011,    pp. 3355-3360

The above cited publication references are incorporated herein byreference for all purposes.

What is claimed is:
 1. An apparatus, comprising: a light detection andranging (LIDAR) sensor configured to detect light; a scan module havinga micro-electromechanical system (MEMS) mirror configured to deflect oneor more laser beams to project vector graphic content; a unifiedprocessor configured to receive information from the LIDAR sensor todetermine a physical property of an object or an environment and toproject vector graphic content using the scan module, wherein theunified processor reduces processing delays by fusion of processing ofboth determination of the physical property of the object or theenvironment and vector graphic content; a mechanical mounting for boththe scan module and the LIDAR sensor wherein the scan module and theLIDAR sensor are attached to the mechanical mounting.
 2. The apparatusof claim 1 wherein the fusion of processing of both the vector graphicscontent and the determination of the physical property of the object orthe environment is performed at a firmware layer of the unifiedprocessor.
 3. The apparatus of claim 1 wherein the fusion of processingof both the vector graphics content and the determination of thephysical property of the object or the environment is performed at asoftware layer of the unified processor.
 4. The apparatus of claim 1wherein the mechanical housing further includes the unified processorwhereby a physical proximity of the unified processor, LIDAR sensor andthe scan module further reduces processing delays of the determinationof the physical property of the object or the environment and vectorgraphics content.
 5. The apparatus of claim 1 wherein the LIDAR sensoris an optical time of flight sensor.
 6. The apparatus of claim 1 whereinthe LIDAR sensor is a Frequency Modulated Continuous Wave optical timeof flight sensor. 7 The apparatus of claim 1 wherein the LIDAR sensor isan Amplitude Modulated Continuous Wave optical time of flight sensor. 8.The apparatus of claim 1 wherein the LIDAR sensor is a pulsed opticaltime of flight sensor.
 9. The apparatus of claim 1 wherein the LIDARsensor is an infrared sensor.
 10. The apparatus of claim 1 wherein thephysical property of the object is a 3 dimensional position of theobject.
 11. The apparatus of claim 1 wherein the physical property ofthe object is a speed of the object.
 12. The apparatus of claim 1wherein the object is a floor or a wall.
 13. The apparatus of claim 1wherein the processor is part of a single board computer.
 14. Theapparatus of claim 1 further comprising one or more additional scanmodules configured to project additional vector graphic content.
 15. Theapparatus of claim 1 wherein the LIDAR sensor detects light from a Fieldof View (FOV) and the apparatus further comprises one or more additionalLIDAR sensors configured to detect light from an additional FOV.
 16. Theapparatus of claim 1 further comprising an auxiliary sensor wherein theauxiliary sensor is coupled to the mechanical mounting.
 17. Theapparatus of claim 1 wherein a projection area of the vector graphiccontent overlaps a FOV of the LIDAR sensor.
 18. The apparatus of claim 1wherein processing of the vector graphics content depends upon thedetermination of the physical property of the object or the environment.19. The apparatus of claim 1 wherein the physical property of the objectis a speed of the object traveling from the apparatus and the projectionof the vector graphics content includes an indication of the speed ofthe apparatus.
 20. The apparatus of claim 1 wherein the physicalproperty of the object is a distance of the object from the apparatusand the vector graphics content includes the distance of the object fromthe apparatus.
 21. An apparatus, comprising: a light detection andranging (LIDAR) sensor configured to detect light and sense a physicalproperty of an object or an environment; a scan module having amicro-electromechanical system (MEMS) mirror configured to deflect oneor more laser beams to project vector graphic content related to thephysical property of the object or the environment; a unified processorconfigured to reduce processing delays by fusion of processing of bothdetermination of the physical property of the object or the environmentand vector graphic content; and a mechanical mounting for both the scanmodule and the LIDAR sensor wherein the scan module and the LIDAR sensorare attached to the mechanical mounting.
 22. The apparatus of claim 21wherein the physical property is a distance of the object.
 23. Theapparatus of claim 21 wherein the physical property is a proximity ofthe object.
 24. The apparatus of claim 21 wherein the physical propertyis a velocity or acceleration of the object or of the apparatus relativeto the environment.
 25. The apparatus of claim 21 wherein the physicalproperty is a presence of the object.
 26. The apparatus of claim 21wherein the physical property is at least one of 3 dimensional shapemeasurements and 3-dimensional orientation measurements.
 27. Theapparatus of claim 21 wherein the physical property is a presence of ahuman in the environment.
 28. The apparatus of claim 21 wherein thephysical property is a facial features and the object is a human face,wherein determination of the physical property of the object or theenvironment includes facial recognition.
 29. The apparatus of claim 21wherein determination of the physical property of the object or theenvironment includes recognition of the object.
 30. The apparatus ofclaim 21 wherein the LIDAR sensor is further configured to detectvisible light wavelengths.
 31. The apparatus of claim 21 wherein theLIDAR sensor is further configured to detect light at infraredwavelengths.
 32. The apparatus of claim 21 wherein the LIDAR sensor isfurther configured to sense the physical property of the object orenvironment from the detected light.
 33. The apparatus of claim 21wherein the light is one or more of the one or more laser beams.
 34. Theapparatus of claim 21 wherein the LIDAR sensor is an optical time offlight sensor.
 35. The apparatus of claim 21 wherein the LIDAR sensor isan Amplitude Modulated Continuous Wave optical time of flight sensor.36. The apparatus of claim 21 wherein the LIDAR sensor is a pulsedoptical time of flight sensor.
 37. The apparatus of claim 21 wherein theLIDAR sensor is a Frequency Modulated Continuous Wave (FMCW) opticaltime of flight sensor. .
 38. The apparatus of claim 21 wherein the LIDARsensor is configured to sense the physical property of the object orenvironment within a field of view and the apparatus further comprisingone or more additional sensors wherein the one or more additionalsensors are configured to expand the field of view for sensing thephysical property of the object or environment.
 39. The apparatus ofclaim 38, wherein the LIDAR sensor and the one or more additionalsensors are arranged in an array configuration.
 40. The apparatus ofclaim 21 wherein the LIDAR sensor has bandwidth to sense the physicalproperty of the object or environment and the apparatus furthercomprising one or more additional sensors wherein the one or moreadditional sensors are configured to increase the bandwidth to sense thephysical property of the object or environment.
 41. The apparatus ofclaim 40, wherein the LIDAR sensor and the one or more additionalsensors are arranged in an array configuration
 42. The apparatus ofclaim 21 further comprising at least one additional sensor configured tosense a different physical property of the object or environment. 43.The apparatus of claim 42, wherein the LIDAR sensor and the one or moreadditional sensors are arranged in an array configuration
 44. Theapparatus of claim 21 wherein the scan module has an FOV for projectingvector graphic content related to the physical property of the object orthe environment and the apparatus further comprising at least oneadditional scan module configured to increase the FOV for projectingvector graphic content.
 45. The apparatus of claim 44, wherein the scanmodule and the one or more additional scan modules are arranged in anarray configuration
 46. The apparatus of claim 21 wherein the one ormore lasers have a laser light wavelength bandwidth for projectingvector graphic content related to the physical property of the object orthe environment and the apparatus further comprising one or moreadditional scan modules configured to increase the range laser lightwavelength of bandwidths for projecting vector graphic content.
 47. Theapparatus of claim 46, wherein the scan module and the one or moreadditional scan modules are arranged in an array configuration.
 48. Theapparatus of claim 21 further comprising one or more additional scanmodules configured to display additional vector graphic content.
 49. Theapparatus of claim 48, wherein the scan module and the one or moreadditional scan modules are arranged in an array configuration.
 50. Theapparatus of claim 21 wherein the LIDAR sensor and scan module arearranged such that their apertures are fully coaxial and an FOV foreither of them can be adjusted, aligned, and/or expanded by commonlyshared optics.